Systems and methods for de-blurring motion blurred images

ABSTRACT

Systems and methods for providing a substantially de-blurred image of a scene from a motion blurred image of the scene are disclosed. An exemplary system includes a primary detector for sensing the motion blurred image and generating primary image information representing the blurred image, a secondary detector for sensing two or more secondary images of the scene and for generating secondary image information representing the two or more secondary images, and a processor for determining motion information from the secondary image information, estimating a point spread function for the motion blurred image from the motion information, and applying the estimated point spread function to the primary image information to generate information representing the substantially de-blurred image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/US03/19446, filed Jun. 18, 2003, published on Dec. 31, 2003 asInternational Patent Publication No. WO 04/001667, which claims priorityto U.S. Provisional Patent Application No. 60/390,336, filed on Jun. 21,2002, which is incorporated herein by reference for all purposes andfrom which priority is claimed.

NOTICE OF GOVERNMENT RIGHTS

The United States government has certain rights in the present inventionpursuant to National Science Foundation ITR Award IIS-00-85864.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to techniques for digitally capturing andprocessing still images of a scene using an image detector, and moreparticularly, to techniques for correcting blurring introduced into suchimages due to motion of the image detector.

2. Background Art

Motion blur due to camera shake is a common problem in photography,especially in conditions involving zoom and low light. Merely pressing ashutter release button on the camera can in and of itself cause thecamera to shake, and unfortunately cause blurred images. This problem isespecially prevalent in digital photography, where lightweight cameraswith automated exposure times are not easily stabilized are common, andwhere automated exposure times often necessitate relatively longstabilization to ensure a non blurred image. The compact form and smalllenses of many of these cameras only serves to increase this problem.

The sensor of a digital camera creates an image by integrating energyover a period of time. If during this time—the exposure time—the imagemoves, either due to camera or object motion, the resulting image willexhibit motion blur. The problem of motion blur is increased when a longfocal length (zoom) is employed, since a small angular change of thecamera creates a large displacement of the image, and in situations whenlong exposure is needed, either due to lighting conditions, or due tothe use of small aperture.

There have been several attempts to provide a solution to this problem.One group of solutions focus on the reduction or elimination of relativemovement between the camera and the scene during the integration time.Such solutions include the use of a tripod, flash photography, the useof increased sensor sensitivity, the use of an increased aperture, anddynamic image stabilization.

A stable tripod that can resist wind, and a shutter release cable thatdoes not transmit hand vibration to a camera mounted on such a tripod,eliminates the problem of camera shake where both the mounted camera andscene are static. One limitation is that only professionals are likelyto use a tripod and shutter release cable on a regular basis. Moreover,the use of a tripod does not solve the problem of shooting from a movingplatform, such as car, train, helicopter or balloon.

A photographic flash produces a strong light flux that sustained for afraction of a section (less than 1/1000). The exposure time is set tobracket the flash time (usually 1/60 sec), and the aperture of thecamera is set to match the flash flux. Therefore, blur caused by motionduring the bracket time has very low intensity. In essence, the flash“freezes” motion of both camera and moving objects. However, objects inbright daylight may still have motion blur and, of course, flashphotography is useful only if the distance between the flash and theobject is small.

Increasing the sensor sensitivity, and therefore reducing the exposuretime, can decrease the problem of motion blur. However, it cannoteliminate blur completely. Moreover, Sensitive sensors (both film andCCD) produce noisy and grainy images.

Increasing the aperture size greatly decreases the required exposuretime, and thus reduces motion blur. Unfortunately, cost and weight alsosignificantly increase with an increased lens aperture, and a tripod maybe required to comfortably handle such weight. Also, the use of a largeraperture lens is applicable only for more expensive cameras where it ispossible to replace the lens.

In addition, the use of dynamic image stabilization involves theincorporation of inertial sensors, such as gyroscopes, to sense andcompensate for camera shake in real time by moving an optical element.While this technology is used in stabilized zoom lens for Single LensReflex (“SLR”) cameras, it is costly, and its effectiveness is limitedto approximately 1/60 of a second for typical 400 mm zoom lens. Thesensitivity of such system to very slow motion may also be limited, andmay suffer from drift. In addition, such system cannot compensate forconstant speed motion, such as occurs when taking images from a movingtrain.

Accordingly, while addressing the problem of motion of the camera itselfis useful in certain applications, it does not provide an adequatesolution to the problem of motion blur as such systems are eitherlimited, very costly, or both. An alternative approach is to correctblur after the image has been taken by using a de-blurring algorithm.

However, while approaches which either assume that the point spreadfunction is known or can be modeled by a simple function and foundautomatically from the image itself, no satisfactory solutions have thusfar been provided. In particular, it has been difficult to obtain auseful point spread function useful in a de-blurring algorithm sinceinaccurate point spread functions tends to create strong artifacts,making them unpleasant for the eye. Accordingly, there remains a needfor a technique for correcting blurring introduced into an image due tocamera motion by finding an accurate point spread function.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a technique forcorrecting blurring introduced into an image due to camera motion.

A further object of the present invention is to provide a technique forcorrecting blurring introduced into an image due to camera motion byusing associated motion information.

Another object of the present invention is to provide an apparatus ableto capture all information required to correct blurring introduced intoan image due to camera motion.

Still another object of the present invention is to provide an apparatusfor capturing a motion blurred image, de-blurring the image, andproviding a user with a de-blurred image.

In order to meet these and other objects of the present invention whichwill become apparent with reference to further disclosure set forthbelow, the present invention discloses a system for providing asubstantially de-blurred image of a scene from a motion blurred image ofthe scene. The system includes a primary detector for sensing the motionblurred image at a first predetermined resolution and generating primaryimage information representing the blurred image, a secondary detectorfor sensing two or more secondary images of the scene and for generatingsecondary image information representing the two or more secondaryimages, and a processor. The processor is advantageously adapted todetermine motion information from the secondary image information,estimate a point spread function for the motion blurred image from themotion information, and apply the estimated point spread function to theprimary image information to generate information representing thesubstantially de-blurred image.

In one arrangement, the system includes a first camera housing theprimary detector, a second camera housing the secondary detector, and arigid member connecting the cameras. Alternatively, a single camera mayhouse both the primary and secondary detectors.

In another preferred arrangement, a beam splitter having one input areaand first and second output areas is provided. The beam splitter isoptically coupled to the scene at the input area, to the primarydetector at the first output area, and to the secondary detector at thesecond output area. Advantageously, the beam splitter may be anasymmetric beam splitter adapted to output greater than 50% of an inputimage energy through the first output area, and preferably approximately90% of an input image energy through the first output area.

In still another preferred arrangement, the primary detector is a firstportion of a dual-resolution sensor and the secondary detector a secondportion of the dual-resolution sensor. The ratio of the firstpredetermined resolution to said second predetermined resolution ispreferably 9:1 in terms of the scene energy incident on the sensor. Thetwo portions may advantageously be formed on a single chip, to ensure alow cost and compact system.

The present invention also provides methods for providing asubstantially de-blurred image of a scene from a motion blurred image ofsaid scene. In one method, the motion blurred image of the scene and twoor more secondary images are sensed. Next, primary image informationrepresenting the blurred image and secondary image informationrepresenting the two or more secondary images are generated, and motioninformation from the secondary image information is determined. A pointspread function for said motion blurred image from said motioninformation; and the estimated point spread function is applied to theprimary image information to generate information representing thesubstantially de-blurred image

Advantageously, fifteen or more secondary images of the scene should besensed at the second predetermined resolution. It is preferred thatglobal motion information is determined from the secondary imageinformation, and a continuous point spread fiction estimated from theglobal motion information.

The accompanying drawings, which are incorporated and constitute part ofthis disclosure, illustrate preferred embodiments of the invention andserve to explain the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1( a)-(c) are block diagrams of exemplary systems in accordancewith the present invention;

FIG. 2 is a graph showing the relationship between temporal resolutionand spatial resolution;

FIG. 3 is a graph showing an illustrative point spread function;

FIGS. 4( a)-(d) are graphs illustrating the computation of a continuouspoint spread function from discrete motion vectors in accordance with anembodiment of the present invention;

FIGS. 5( a) and (b) are a flow diagrams of a method for de-blurring amotion blurred image including deriving a point spread function inaccordance with the present invention;

FIGS. 6( a)-(d) are illustrative diagrams showing exemplary tools thatmay be used to model a point spread function in accordance with analternative embodiment of the present invention;

FIG. 7 is a flow diagram of a method for determining a point spreadfunction in accordance with the embodiment of FIGS. 6( a)-(d);

FIGS. 8( a)-(b) are illustrative diagrams showing exemplary method formeasuring a point spread function in accordance with another alternativeembodiment of the present invention;

FIG. 9 is a graph of an exemplary estimated point spread function; and

FIGS. 10( a)-(c) are exemplary images of a scene.

Throughout the Figs., the same reference numerals and characters, unlessotherwise stated, are used to denote like features, elements, componentsor portions of the illustrated embodiments. Moreover, while the presentinvention will now be described in detail with reference to the Figs.,it is done so in connection with the illustrative embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIGS 1(a)-(c), exemplary hybrid camera systems inaccordance with the present invention are shown. Each system includes aprimary image detector for capturing an image of the scene, as well as asecondary detectors for capturing information useful for correctingblurring introduced into the image due to camera motion.

The embodiment shown in FIG. 1( a) uses two cameras 101, 103 connectedby a rigid member 105. Camera 101 is preferably a high-resolution stillcamera, and includes the primary detector 102 to capture an image of ascene. Camera 103 is preferably a low-resolution video camera whichincludes a secondary detector 104.

The secondary detector 104 is used for obtaining motion information, andtherefore must capture a minimum of two frames of digital videoinformation in order to provide such motion information. Preferably,fifteen or more frames are captured during the integration time ofcamera 101. While the embodiment shown with reference to FIG. 1( a)depicts a camera mounted on a camera, other two camera arrangements,such as a camera within a camera, may be utilized to achieve the sameresults.

The detector 102 may be traditional film, a CCD sensor, or CMOS sensor.Secondary detector 104 may likewise be a CCD or CMOS sensor. It isadvantageous for the secondary detector 104 to be a black and whitesensor, since such a detector collects more light energy (broaderspectrum) and therefore can have higher temporal resolution. Inaddition, since the secondary detector is used only as a motion sensor;it can be of low spatial resolution to increase the temporal resolutionand have high gain.

An alternative embodiment, shown in FIG. 1( b), employs a single camera110 and a beam splitter 111 to generate two image paths leading toprimary 112 and secondary 113 detectors. This system requires lesscalibration than the system of FIG. 1( a) since the same camera lens isshared, and hence results in identical image projection models. Thoseskilled in the art will recognize that a tradeoff exists between thespatial resolution of the primary detector 112 and the ability toaccurately provide motion information by the secondary detector 113 asthe beam splitter 111 divides the available energy between the twodetectors 112, 113. Therefore, while a beam splitter which divides theenergy 50/50 between the two detectors may be utilized, it is preferredthat the division be greater than 80/20, for example, an approximately90/10 split, with more energy going to the primary 112 detector.

In a highly preferred arrangement, the beam splitter 111 is anasymmetric beam splitter that passes most of the visible light to theprimary detector 112 and reflects non-visible wavelengths toward thesecondary detector 111. For example a “hot mirror” beam splitter whichis commercially available from Edmund Industrial Optics may be employed.

Another alternative embodiment, shown in FIG. 1( c), uses a special chip121 that includes both primary and secondary detectors. The chip 121includes both a high-resolution central area 125, which functions as theprimary detector, and a low resolution peripheral areas 126, 127, whichfunction as the secondary detector.

The chip 121 may be implemented using binning technology now commonlyfound in CMOS and CCD sensors. Binning allows the charge of a group ofadjacent pixels to be combined before digitization. This enables thechip to switch between a normal full-resolution mode, when binning isnot used, and a hybrid primary-secondary detector mode, when binning isactivated. In the hybrid mode, the primary detector portion 125 capturesa high resolution image, while the secondary detector portions 126, 127capture a sequence of low resolution images from which motioninformation can be derived.

Given sufficient light, CCD and CMOS sensors can detect a scene at veryfast rates and thereby avoid camera blur. However, motion blur willoccur when there is not sufficient light for fast imaging, since theamount of energy reaching each pixel is

${\int{\frac{\Psi_{fov} \cdot k}{n}{\mathbb{d}t}}},$where: ψ_(fov) is the flux though the field of view, k is the fillfactor, n is the number of pixels, and the integral is taken overexposure time. Reducing the number of pixels while keeping the samefield of view equates into lowering resolution, and thereby increasingthe energy per pixel.

Lower resolutions may be achieved either by using a low-resolution chip,or through binning, as discussed above. Examples for the image dimensionof the hi-resolution and low-resolution sensors for example ratios of1/36 and 1/64 pixels at common resolutions are given in Table 1.

TABLE 1 Hi res 1024 × 768 1280 × 960 1600 × 1200 2048 × 1536 2560 × 19201/36 ratio 170 × 128 213 × 160 266 × 200 341 × 256 426 × 320 1/64 ratio128 × 96  160 × 120 200 × 150 256 × 192 320 × 240

Also as shown in FIG. 1( c), the camera 120 preferably includes acircuit card or area which includes memory 121 for storing both theprimary and secondary images sensed by detector portions 125-127. Thecamera also preferably includes processor 122 for computing motion fromthe sequence of low resolution images, estimating a point spreadfunction for the primary image from such motion information, andde-blurring the primary image with the estimated point spread functionby applying a deconvolution algorithm, each of which are describedbelow. The processor and memory should be sufficiently small to beimplemented within the camera. Exemplary software that may be stored inmemory 121 and executed on processor 122 is included herein as AppendixA. The de-blurred image may then be displayed to the operator of camera120 threw a standard display (not shown), or stored for later use.

Alternatively, the motion computation, point spread function estimation,and de-blurring functions may be performed by a separate computer, sucha s personal computer running the software of Appendix A. In addition,while the foregoing description has been with respect to the embodimentshown in FIG. 1( c), it equally applies to the embodiments shown inFIGS. 1( a) and (b), as each may be readily modified to include suitablememory and processing capacity. Likewise, the software of Appendix A isexemplary, and alternative software arrangements in a variety ofprogramming languages may be utilized for performing such functionality.

Referring next to FIG. 2, a graph illustrating the fundamental tradeoffbetween spatial resolution and temporal resolution in an imaging systemis shown. An image is formed when light energy is integrated by an imagedetector over a time interval. Assuming that the total light energyreceived by a pixel during integration must be above a minimum level forthe light to be detected, the minimum level is determined by thesignal-to-noise characteristics of the detector. Therefore, given such aminimum level and an incident flux level, the exposure time required toensure detection of the incident light is inversely proportional to thearea of the pixel. In other words, exposure time is proportional tospatial resolution. When the detector is linear in its response, theabove relationship between exposure and resolution is also linear.

The parameters of the line shown in FIG. 2 are determined by thecharacteristics of the materials used by the detector and the incidentflux. Different points on the line represent cameras with differentspatio-temporal characteristics. For instance, a conventional videocamera 210 has a typical temporal resolution 30 fps and a spatialresolution of 720×480 pixels. Instead of relying on a single point, twovery different operating points on the line may be used tosimultaneously obtain very high spatial resolution with low temporalresolution 220 and very high temporal resolution with low spatialresolution 230. This type of hybrid imaging provides the missinginformation needed to de-blur images with minimal additional resources.

Referring next to FIG. 3., an exemplary point spread function is shown.The complete point spread function of a motion-blurred image consists oftwo parts. First and most importantly, there is a point spread functiondue to motion. The derivation of such a point spread function isaddressed in detail below. However, it should be noted that there may bea second component to a complete point spread function, that of theimaging system itself, and may either be measured or modeled usinga-priori knowledge of the imaging system. Those skilled in the art willappreciate that various techniques exist to conduct such measurement ormodeling.

In order to determine the point spread function due to motion, asecondary detector provides a sequence of images (frames) that are takenat fixed intervals during the exposure time. By computing the globalmotion between these frames, samples of the continuous motion pathduring the integration time may be obtained. The motion betweensuccessive frames is limited to a global rigid transformation model.However, the path, which is the concatenation of the motions betweensuccessive frames, is not restricted and can be very complex.Accordingly, the motion between successive frames may be determinedusing a multi-resolution iterative algorithm that minimizes thefollowing optical flow based error function:

$\begin{matrix}{\arg\;{\min_{u,\upsilon}{\sum( {{u\frac{\partial I}{\partial x}} + {v\frac{\partial I}{\partial y}} + \frac{\partial I}{\partial t}} )^{2}}}} & (1)\end{matrix}$where the partial derivatives are the spatial and temporal partialderivatives of the image, and (u, v) is the instantaneous motion at timet. This motion between the two frames is defined by the following globalrigid motion model:

$\begin{matrix}{\begin{bmatrix}u \\v\end{bmatrix} = {\begin{bmatrix}{\cos\;\theta} & {\sin\;\theta} & {tx} \\{{- \sin}\;\theta} & {\cos\;\theta} & {ty}\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}} & (2)\end{matrix}$where (t_(x), t_(y)) is the translation vector and θ is the rotationangle about the optical axis.

Note that the secondary detector, which has a short but nonzerointegration time, may also experience some motion blur. This motion blurcan violate the constant brightness assumption, which is used in themotion computation. However, under certain symmetry conditions, thecomputed motion between two motion blurred frames is the center ofgravity of the instantaneous displacements between these frames duringtheir integration time.

The discrete motion samples that are obtained by the motion computationneed to be converted into a continuous point spread function. For thispurpose, the constraints that a motion blur point spread fiction mustsatisfy are defined and then used in order to estimate the appropriatepoint spread fiction.

Any point spread function is an energy distribution function, which canbe represented by a convolution kernel k: (x, y)=>w, where (x, y) is alocation and w is the energy level at that location. The kernel k mustsatisfy the following energy conservation constraint:∫∫k(x, y)dxdy=1  (3)which states that energy is neither lost nor gained by the blurringoperation (k is a normalized kernel). In order to define additionalconstraints that apply to motion blur point spread functions, a timeparameterization of the point spread fiction is used as a path functionf: t=>(x, y) and an energy function h: t=>w. Due to physical speed andacceleration constraints, f(t) should be continuous and at least twicedifferentiable, where f′(t) is the speed and f″(t) is the accelerationat time t.

By assuming that the scene radiance does not change during imageintegration, an additional constraint is determined:

$\begin{matrix}{{ {{{\int_{t}^{t + {\delta\; t}}{{h(t)}\ {\mathbb{d}t}}} = \frac{\delta\; t}{t_{end} - t_{start}}},{\delta\; t}} \rangle 0},{t_{start} \leq t \leq {t_{end} - {\delta\; t}}},} & (4)\end{matrix}$where [t_START, t_END] is the image integration interval. Thisconstraint states that the amount of energy which is integrated at anytime interval is proportional to the length of the interval.

Given these constraints and the motion centroid assumption, a continuousmotion blur point spread function may be estimated from discrete motionsamples, as illustrated in FIGS. 4( a)-(d).

First, the path f(t) may be estimated by Spline interpolation, as shownin FIGS. 4( a) and (b). Spline curves are preferably used because oftheir smoothness and twice differentiability properties, which satisfythe speed and acceleration constraints.

In order to estimate the energy function h(t), the extent of each framealong the interpolated path must be determined. This may be accomplishedusing the motion centroid assumption by splitting the path f(t) intoframes with a Voronoi Tessellation, as shown in FIG. 4( b).

Since the constant radiance assumption implies that frames with equalexposure times integrate equal amount of energy, h(t) may be computed,up to scale, for each frame as shown in FIG. 4( c). Note that all therectangles in this figure have equal areas.

Finally, h(t) is normalized in order to satisfy the energy conservationconstraint and smooth it. The resulting point spread function is shownin FIG. 4( d). The end result of the above procedure is a continuousmotion blur point spread function that can now be used for motionde-blurring.

Given the estimated point spread function, the high-resolution imagethat was captured by the primary detector may be de-blurred using wellknown image deconvolution algorithms, such as the Richardson Lucyalgorithm. Since this is the only step that involves high-resolutionimages, it dominates the time complexity of the method, which is usuallythe complexity of a Fast Fourier Transform (“FFT”).

Referring next to FIG. 5( a), the foregoing techniques are implementedin a methodology as follows. First, primary image informationrepresenting the blurred image sensed by the primary detector 510, andsecondary image information representing a sequence of images sensed bythe secondary detector 511, are obtained. A standard motion analysisalgorithm is used to determine discrete motion information 520 of theprimary detector. Next, the point spread function for the motion blurredimage is estimated 530 using the discrete motion information. That pointspread function 535 may optionally be convolved with an estimated ormeasured point spread function for the optical system itself 540. Thepoint spread function is then applied to the primary image informationin a standard de-blurring algorithm, 550, and a de-blurred image isoutput 560.

FIG. 5( b) shows the preferred details of the point spread functionestimation step 530. Two-dimensional Spline interpolation is used toprovide a continuous two-dimensional path 531. Voronoi Tessellation isthen used to provide frame partitioning of the interpolatedtwo-dimensional path 532. Equal area rectangles are constructed withineach partitioned fame to determine the mean energy at each frame 533.Finally, the determined mean energy values are smoothed and normalized534.

Referring next to FIGS. 6( a)-(d), exemplary tools that may be used tomodel a point spread function in accordance with an alternativeembodiment of the present invention are shown. In some cases, a pointspread function can be estimated directly from an image itself, withoutthe need for additional motion information. For example, a small brightpoint light source on a dark background, such as dark night, whichhappen to be at the right depth, if the camera was translating, or ifthe point light source was at arbitrary depth (bright star in clear sky)and camera was rotating with no translation—then the image of this pointlight source provide point spread function which is good enough forde-blurring, as long as the dynamic range of the camera is sufficient.

It is unlikely to expect such luck to happen, especially if the cameramotion included translation about the optical axis, since at least twosuch points are needed. Instead, a set of primitives may be createdthat, if found in the image, can help estimating the point spreadfunction. The greatest advantage of this approach is that it does notrequire any additions to the imaging process itself and it can be usedfor existing pictures as well. The disadvantage is that this methodrelies on user skills to estimate the real shape of an object from ablurred image—or from a different image that may not be blurred that wastaken at a different time or different angle.

FIG. 6( a) illustrates an exemplary point tool 610 that may be used todefine an object point. Since object points may not be perfect smallwhite points over a black background, the tool provides means to definepoint size, eccentricity, orientation and color. Point size 611 isselected by a slider or by entering size in pixels of a fraction of apixel. Point eccentricity 612 is selected by a slider or entered as anumber as the ratio between main axes. Regarding orientation 613, ifpoint eccentricity is not 1, then the orientation can be entered using adial or as a number (angle). Point color 614 is selected using colortool, or sampled from the image itself and optionally modified. Inaddition, a background color may be selected using color tool, orsampled from the image itself and optionally modified.

FIG. 6( b) illustrates an exemplary line tool 620 that may be used todefine an object line. The line attributes may include thickness,orientation, and color. Line thickness 621 is selected by a slider, orby entering thickness in pixels (can be fraction of a pixel). Lineorientation 622 can is entered using a dial or as a number (angle). Linecolor 623 is selected using color tool—or sampled from the image itselfand optionally modified. Again, a background color may be selected usingcolor tool, or sampled from the image itself and optionally modified.

FIG. 6( c) illustrates an exemplary ramp tool 630 that may be used todefine a ramp or edge. The attributes may include orientation and color.Ramp orientation 631 can is entered using a dial or as a number (angle).Ramp color 63 may be selected using color tool, or sampled from theimage itself and optionally modified. A background color may be selectedusing color tool, or sampled from the image itself and optionallymodified.

FIG. 6( d) illustrates an exemplary corner tool 640 that may be used todefine an object comer. Comer attributes include angle, orientation, andcolor. The angle 641 may be entered using a dial or as a number (angle).Comer orientation 642 can is entered using a dial or as a number(angle). Corner color 643 may be selected using color tool, or sampledfrom the image itself and optionally modified. A background color may beselected using color tool, or sampled from the image itself andoptionally modified.

For example, a user may download a motion-blurred image from theInternet, and desire to de-blur that image. Using the comer tool of FIG.6( d), the user may examine a small region in the blurred image, e.g.,30×30 pixels, and create a model 30×30 pixel image of what that regionshould look like when de-blurred. That model image, convolved by anunknown point spread function, will equal the original region of theblurred image.

One approach to finding this block is by using a Fourier transform. TheFourier transform of the model image region multiplied by the Fouriertransform of the point spread function is equal to the Fourier transformof the captured image region. Therefore, the Fourier transform of thepoint spread function is determined by dividing the Fourier transform ofthe captured image region by the Fourier transform of the model imageregion, and an inverse Fourier transform may be used to obtain anestimated point spread function of the blurred image. Once obtained, theuser may de-convolve the blurred image with the estimated point spreadfiction to obtain an estimate of the captured imaged. The user then canthen compare the estimated captured image with the original capturedimage, and visually determine whether the further refinements arenecessary.

Referring next to FIG. 7, a method for interactively estimating a pointspread function using the tools of FIGS. 6( a)-(d) is shown. A userselects and classifies features within the blurred image 720, e.g.,using the tools of FIGS. 6( a)-(b).

A common point spread fiction is then determined using de-convolution,where the function is the unknown variable 730. The motion blurred imageis de-blurred using the recovered point spread function 740, and theuser is permitted to view the de-blurred image and refine his or herclassifications as appropriate 750, 755. Finally, the de-blurred imageis output 760.

Referring next to FIGS. 8( a)-(b), an exemplary method for measuring apoint spread function in accordance with another alternative embodimentof the present invention will be described. Laser guides are common inadaptive optics to provide a reference point for wavefront measurements.Similar techniques may be useful for motion de-blurring by projecting areference point on the object using a laser beam as seen in FIG. 8( a).An alternative approach is to attach a (limited) stabilized laser to thecamera itself as shown in FIG. 8( b).

As shown in FIGS. 8( a) and (b), a laser guide is attached to a camera.The laser is mounted on rings (gimbals), which are stabilized usinggyroscopes. The laser emits one or more reference points to the scene.The image of these points is then utilized, after normalization tosatisfy an energy constraint, to determine the point spread functionthat is sought. It should be noted that other light beam sources, suchas columnated light beam sources, may be used in place of a laser.

It should be noted that the practical use of this embodiment is limited,as transmitting a laser beam into living objects may not be practical.However, the technique is useful for natural or commercial photography.

FIG. 9 is a graph of an exemplary estimated point spread functionderived using the apparatus of FIG. 1( a) and the method described inconnection with FIG. 8. FIGS. 10( a)-(c) are exemplary images of thecorresponding scene, which FIG. 10( a) showing an image of the scenetaken from a tripod-mounted camera, FIG. 10( b) showing the blurredimage, and FIG. 10( c) showing the image after de-blurring. While thede-blurred image reveals some artifacts from the de-blurring process, itis a vast improvement over the blurred image. Accordingly, techniquesfor correcting blurring introduced into an image due to camera motionhave been provided

The foregoing merely illustrates the principles of the invention.Various modifications and alterations to the described embodiments willbe apparent to those skilled in the art in view of the teachings herein.For example, the foregoing techniques may be applied to video sequencestaken by a moving camera, using an assumption of symmetricaldistribution to recover the average motion vectors from the motionblurred images. It will thus be appreciated that those skilled in theart will be able to devise numerous systems and methods which, althoughnot explicitly shown or described herein, embody the principles of theinvention and are thus within the spirit and scope of the invention.

1. A system for providing a substantially de-blurred image of a scenefrom a motion blurred image of said scene, comprising: (a) a primarydetector for sensing said motion blurred image of said scene at a firstpredetermined resolution and generating primary image informationrepresenting said blurred image; (b) a secondary detector, coupled tosaid primary detector, for sensing two or more secondary images of saidscene at a second predetermined resolution which is less than or equalto said first predetermined resolution while said motion blurred imageis sensed by the primary detector, and for generating secondary imageinformation representing said two or more secondary images; and (c) aprocessor, coupled to said primary detector and receiving said primaryimage information therefrom, and coupled to said secondary detector andreceiving said secondary image information therefrom, adapted todetermine motion information from said secondary image information,estimate a point spread function for said motion blurred image from saidmotion information, and apply said estimated point spread function tosaid primary image information to thereby generate informationrepresenting said substantially de-blurred image.
 2. The system of claim1, further comprising a first camera housing said primary detector, asecond camera housing said secondary detector, and a rigid memberconnecting said first camera to said second camera.
 3. The system ofclaim 1, further comprising a camera housing said primary and saidsecondary detectors.
 4. The system of claim 3, further comprising a beamsplitter having one input area and first and second output areas, saidbeam splitter being optically coupled to said scene at said input area,optically coupled to said primary detector at said first output area,and optically coupled to said secondary detector at said second outputarea.
 5. The system of claim 4, wherein said beam splitter comprises anasymmetric beam splitter adapted to output greater than 50% of an inputimage through said first output area.
 6. The system of claim 5, whereinsaid beam splitter is adapted to output greater than 80% of an inputimage through said first output area.
 7. The system of claim 4, whereinsaid beam splitter is adapted to output substantially all visible lightcorresponding to an input image through said first output area, and atleast some non-visible electromagnetic radiation corresponding to aninput image through said second output area.
 8. The system of claim 3,wherein said primary detector comprises a first portion of adual-resolution sensor, and said secondary detector comprises a secondportion of said dual-resolution sensor.
 9. The system of claim 1,wherein said first predetermined resolution is at least twice as greatas said second predetermined.
 10. A method for providing a substantiallyde-blurred image of a scene from a motion blurred image of said scene,comprising the steps of: (a) sensing said motion blurred image of saidscene at a first predetermined resolution using a primary detector; (b)sensing two or more secondary images of said scene at a secondpredetermined resolution with a secondary detector which is less than orequal to said first predetermined resolution while said motion blurredimage is sensed; (c) generating primary image information representingsaid blurred image and secondary image information representing said twoor more secondary images; (d) using a processor, determining motioninformation from said secondary image information; (e) estimating apoint spread function for said motion blurred image from said motioninformation; and (f) applying said estimated point spread function tosaid primary image information to thereby generate informationrepresenting said substantially de-blurred image.
 11. The method ofclaim 10, wherein said step (b) comprises sensing five or more secondaryimages of said scene at said second predetermined resolution, and saidstep (c) comprises generating primary image information representingsaid blurred image and secondary image information representing saidfive or more secondary images.
 12. The method of claim 10, wherein step(d) comprises the step of determining global motion information fromsaid secondary image information.
 13. The method of claim 12, whereinstep (e) comprises the step of estimating a continuous point spreadfunction for said motion blurred image from said global motioninformation.
 14. The method of claim 13, wherein step (f) comprises thestep of deconvolving said primary image information with estimated pointspread function to thereby generate information representing saidsubstantially de-blurred image.
 15. A method for providing asubstantially de-blurred image of a scene from primary image informationrepresenting a motion blurred image of said scene captured at a firstpredetermined resolution using a primary detector, and secondary imageinformation representing two or more secondary images of said scenecaptured at a second predetermined resolution with a secondary detectorwhich is less than or equal to said first predetermined resolution whilesaid motion blurred image is captured, comprising the steps of: (a)using a processor, determining motion information from said secondaryimage information; (b) estimating a point spread function for saidmotion blurred image from said motion information; and (c) applying saidestimated point spread function to said primary image information tothereby generate information representing said substantially de-blurredimage.
 16. The method of claim 15, wherein step (a) comprises the stepof determining global motion information from said secondary imageinformation.
 17. The method of claim 16, wherein step (b) comprises thestep of estimating a continuous point spread function for said motionblurred image from said global motion information.
 18. The method ofclaim 17, wherein step (c) comprises the step of deconvolving saidprimary image information with estimated point spread function tothereby generate information representing said substantially de-blurredimage.